Intrusion prevention systems (IPS) monitor a network for malicious activity, e.g., by blocking and alerting administrators regarding potential network attacks. Deep packet inspection (DPI) methodologies are often used in IPS to match transport layer payload with a database of a predefined pattern which defines an attack (signatures). IPS is sometimes differentiated into host-based intrusion prevention systems (HIPS) and network-based intrusion prevention systems (NIPS) depending on where in the network they are deployed. If the IPS is deployed in an end host, i.e., a target node or source node, then it often is referred to as HIPS: A typical example is an antivirus system found in the end host systems. On the other hand, NIPS is deployed in an intermediate network device like a router whose primary functionality is forwarding the network packets to the right destination port.
In certain scenarios in NIPS, NIPS can accept a packet of a packetized data flow through the network for DPI which the end host rejects (insertion attack); and on the other hand, the NIPS may not process a packet which the end host processes (evasion attack). Insertion and evasion attacks belong to a category of attacks targeting the NIPS, so that the attacks focused on end hosts are not identified by the NIPS. Since NIPS typically use passive monitoring techniques to recognize patterns of signatures in network traffic, attackers can take advantage of ambiguities in network protocol implementations to deceive them. The premise in creating these attacks is to introduce an ambiguity in packet processing before or during DPI between the NIPS and the end host. Insertion and evasion attacks have to be addressed by NIPS due to their passive monitoring of the networks and algorithms to identify these attacks will enhance the quality of DPI in them. Table 1 outlines various ways through which both insertion and evasion attacks can be created by attackers at various layers.
TABLE 1Description of various evasion attack techniquesTechniqueLayerCommentsUncommon>L2NIPS processes a packet with uncommonProtocolprotocol field combination while endFieldsvictim does notObfuscation and>L4Transformation of malicious packetEncryptionpayloads into semantically equivalentones. E.g.: URI hexadecimal encodingDenial ofL2/L3/L4DoS attack intends to overwhelmServicenetwork bandwidth or system resources(DoS)such as CPU or memory byinjecting packetsPacketL3/L4Signature split across multiple packetsSplittingOverlapping packets with signatureTechniquessplitting techniques
Attacks based on techniques like uncommon protocol fields, obfuscation, and encryption-based can often be identified through meticulous packet processing in NIPS. Special consideration has to be given to those algorithms implemented in NIPS in their architecture stages to make them DoS proof.
However, attacks based on packet splitting techniques are quite challenging to identify as packet reassembly is not a primary task of NIPS because of which they can be used to mask other attacks. Thus algorithms which can efficiently identify attacks based on packet splitting techniques are fundamental for NIPS and enhance the quality of signature matching process in DPI.
Packet splitting attacks can be created by attackers in various ways. In simple scenarios, an attack packet with a signature is split into multiple smaller packets and sent out of order so that the NIPS cannot identify the attack. However, in such a scenario, effective reassembly algorithms may be able to identify such an attack. In some complex scenarios, an overlap attack is created by sending duplicate copies of packets representing same IP fragments or TCP segments with different content in the data portion of the packet with only one version of the packet containing a signature or a portion of the signature; here, complete overlapping of the duplicate packets results. Partial overlapping of packets is another way through which evasion attacks are created whereby partial portions of the packets overlap to confuse the NIPS.
Different operating systems reassemble duplicate and overlapping packets in a different way which makes overlap attacks very difficult to process for NIPS. Adequate delays injected by attackers between overlapping or duplicate packets makes the process of detecting these attacks more difficult.
Techniques are known to provide countermeasures against attacks based on overlapping packets (overlap attack).
A detailed overview of security related shortcomings of TCP, IP protocol suite was documented in T. H. Ptalek and T. N. Newsham, “Insertion, Evasion and Denial of Service: Eluding Network Intrusion Detection,” (1998), in which four commercially available NIPS were analyzed. The results showed that fragmentation-based attacks creating overlapping packets are mostly not handled by the NIPS systems resulting in the potential of successful evasion attacks. A detailed analysis of various techniques against overlap attacks is mentioned in T.-H. Cheng and Y.-D. L. Y.-C. L. P.-C. Lin, “Evasion Techniques: Sneaking through you Intrusion Detection/Prevesion systems,” IEEE, Communications Surveys and Tutorials, 14 (2012), 1011-1020. Solutions for overlap attacks have been broadly categorized into flow-modification techniques and operating system (OS) fingerprinting techniques through which networking policies of destination could be identified.
Flow modification techniques typically remove the ambiguity by picking up one interpretation of the packet in case of overlap attacks; other packets are discarded. So-called Protocol Scrubbing, see D. Watson, M. Smart, G. R. Malan and F. Jahanian, “Protocol Scrubbing: network security through transparant flow modification,” IEEE/ACM Transactions on Networking, 12 (2004) 261-273, implements a transport scrubber which picks up one interpretation of the protocols and converts incoming flows into a single representation that all endpoints will universally accept. The scrubber maintains a copy of the byte stream sent by sender, but not acknowledged by the receiver and discards packets that could lead to inconsistencies in the byte stream.
A TCP based overlap detection method is proposed in patent publication U.S. Pat. No. 7,114,181. This implementation introduces the idea of using expected sequence number which is the sequence number of the next in-order packet. If the sequence number of the incoming packet is not the same as the expected sequence number, then the packet is placed in an out-of-order buffer. For every new packet which matches the expected sequence number, a comparison is done with those packets placed in the out-of-order buffer and if an overlap is observed, the packet in the out-of-order buffer is discarded. Both implementations mentioned above need comparably large amounts of memory to store the packets which can be well utilized by the attackers to create memory-based DoS attacks which flood the buffer with data.
To alleviate this problem of high memory needs, M. Vutukuru, H. Balakrishnan and V. Paxson, “Efficient and Robust TCP Stream Normalization,” Proc. IEEE Symposium on Security and Privacy, Oakland (2008) presents an algorithm in which instead of storing the packets directly in the memory, a hash of the payload is computed and stored in the memory and used for further analysis. Even though this is a comparably memory efficient way, processor utilization is very high due to the implementation of hashing techniques.
The second category to address overlap attacks uses fingerprinting techniques to gather information about clients connected to the network so that ambiguity with respect to reassembly can be removed. A method called active mapping was proposed in S. U and P. V, “Active Mapping: resisting NIDS evasion without altering traffic,” in Proc. Symposium on security and privacy (2003). Here, probe packets are sent to client machines connected to NIPS to identify their operating systems and the NIPS can process data depending on the client. But the probing packets can be quite noisy and be blocked by the firewall of the end node victim.
A passive OS fingerprinting method was proposed in G. Taleck, “Ambiguity resolution via passive OS fingerprinting,” in Proc. International conference on recent advances in intrusion detection (2003). Here, passive methods by observing patterns in network protocol header fields are used to identify the OS of the end node victim. The above implementation suffers from a cold start problem and is error prone in certain scenarios where the users may change the default behavior of the OS.
Thus, prior art techniques such as those relying on buffering unacknowledged data and out-of-order data, respectively, often suffer from memory exhaustion and can be a source for DoS attacks. Other techniques require comparably large processing resources for calculating the hash for each and every incoming byte. Fingerprinting techniques can remove ambiguities of data reassembly, but managing information about end nodes can be a tedious process for network administrators. Also, the predictability in processing network data in case of duplicate and overlap packets can be used by attackers along with other evasion techniques mentioned above to create evasion attacks.